Towards autoscaling of Apache Flink jobs

Abstract Data stream processing has been gaining attention in the past decade. Apache Flink is an open-source distributed engine that able to process a large amount of data real time with low latency. Computations are among cluster nodes. Currently, provisioning appropriate cloud resources must be done manually ahead time. A dynamically varying workload may exceed capacity cluster, or leave underutilized. In our paper, we describe architecture enables automatic scaling jobs on Kubernetes based custom metrics, and simple policy. We also measure e ects state size target parallelism duration operation, which considered when designing autoscaling policy, so job respects Service Level Agreement.